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Abstract 

This  report  describes  the  work  done  at  the  University  of  Maryland  in  collaboration  with 
McDonnell  Douglas  Aerospace  on  site  model-based  image  exploitation,  during  the  period 
3/1/95  -  7/31/95.  The  goals  are  (a)  to  use  EO  imagery  obtained  from  a  reconnaissance 
flight  to  build  a  detailed  3-D  model  of  the  target  site,  and  (b)  to  use  the  site  model  to 
register  and  exploit  images  from  an  IR  sequence  obtained  from  a  sensor  mounted  on  a 
platform  approaching  the  target  site.  The  ultimate  aim  is  the  precise  guidance  of  the  platform 
based  on  a  comparison  of  reference  images  generated  from  the  site  model  and  the  IR  images 
obtained  by  the  platform  itself. 


The  support  of  McDonnell  Douglas  Aerospace  under  PO#  Z50108  is  gratefully  acknowledged 


1  Introduction 


Several  military  and  civilian  applications  could  benefit  greatly  from  the  site-model  based 
Image  Exploitation  (IE)  techniques  developed  over  the  past  few  at  the  University  of  Maryland 
(e.g.  [1]).  McDonnell  Douglas  Aerospace  is  interested  in  using  such  techniques  for  the 

problem  of  terminal  guidance  of  a  moving  platform.  The  goal  is  to  enable  a  moving  platform 
to  home  in  on  a  precise  location  inside  a  target  site.  The  platform  is  assumed  to  be  equipped 
with  an  infra-red  (IR)  sensor,  which  continuously  acquires  a  sequence  of  IR  images  of  the 
target  site.  The  idea  is  to  compare  the  images  in  this  sequence  to  “reference”  images  of  the 
target  site  synthesized  from  assumed  camera  positions,  and  thus  help  the  platform  determine 
its  own  position  and  orientation  relative  to  the  target.  The  platform  can  then  make  the 
necessary  adjustments  to  its  motion  in  order  to  stay  on  the  right  trajectory  with  respect  to 
the  target. 

One  way  to  generate  reference  images  is  to  build  a  3-D  model  of  the  site,  and  then 
“render”  the  site  from  the  desired  viewpoint.  The  site  model  can  be  constructed  using  a  set 
of  reconnaissance  images  of  the  site  obtained  by  an  aircraft  flying  around  the  site. 

In  this  report,  we  describe  the  work  done  at  the  University  of  Maryland  in  collaboration 
with  McDonnell  Douglas  Aerospace  during  the  period  3/1/95  -  7/31/95,  as  well  as  the 
work  that  is  proposed  to  be  carried  out  during  the  next  two  years  on  this  project.  We 
have  made  extensive  use  of  the  Radius  Common  Development  Environment  (RCDE)  in  this 
project.  RCDE  provides  a  user-friendly  environment  for  site-model  construction  and  image 
registration.  Details  about  RCDE  can  be  found  in  [5]. 

2  Input  Data 

The  input  data  (listed  in  the  ATG  Reconnaissance  Catalog)  consists  of  the  following: 

(a)  Images  of  the  site  taken  from  a  reconnaissance  aircraft 

(b)  Geocentric  camera  positions:  latitude,  longitude,  and  altitude 

(c)  Target-centered  camera  positions:  range,  azimuth  (heading  angle)  and  elevation  (depres¬ 
sion  angle)  w.r.t  the  target 

(d)  Camera  internal  parameters:  field  of  view  (FOV)  and  focal  length  (f) 
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Figure  1:  Reconnaissance  flight  scenario. 

(e)  Geocentric  coordinates  of  an  aim  point  (target)  in  the  site 

In  addition  to  the  above,  plan  views  and  elevations  of  the  site  were  used  to  obtain  ground 
truth  3-D  locations  of  control  points  in  the  site. 

A  typical  reconnaissance  flight  scenario  is  shown  in  Fig.  1.  This  is  based  on  the  “C” 
sequence  from  the  Labadie  power  plant  data  set.  Typical  reconnaissance  images,  again  from 
the  Labadie  set,  are  shown  in  Fig.  2. 

3  Calculation  of  camera  positions  relative  to  the  target 

This  can  be  done  in  two  ways,  either  using  geocentric  positions  (Fig.  3),  or  using  target- 
centered  positions  (Fig.  4).  In  the  first  case,  all  measurements  of  aircraft/camera  positions 
are  in  absolute  terms,  independent  of  the  site’s  location.  In  the  second,  all  the  measurements 
are  w.r.t.  the  site  under  consideration.  This  provides  a  way  of  cross-checking  the  results.  In 
the  case  of  the  Labadie  data  set,  the  results  tallied  to  within  20  m. 
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Figure  2:  Reconnaissance  images  of  the  Labadie  power  plant. 
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Using  geocentric  coordinates  Consider  two  coordinate  systems  attached  to  the  earth: 


•  The  geocentric  coordinate  system  (GCS):  {Og,Xg,  Yg,  Zg)  with  the  origin  at  the  center 
of  the  earth,  the  Xg  axis  passing  through  the  Greenwich  Meridian,  and  the  Zg  axis 
passing  through  the  north  pole. 

•  The  local  vertical  coordinate  system  (LVCS):  (p.^,Xy,,Y^,Zyj)  with  its  origin  at  some 
chosen  ground  level  point  at  the  target,  and  the  X^,Y^  and  Z^,  axes  corresponding  to 
the  local  north,  local  east  and  the  local  vertical  directions,  respectively. 


The  aim  here  is  to  express  the  camera  position  (specified  in  terms  of  its  latitude,  longitude 
and  altitude)  in  the  LVCS  at  the  site.  We  choose  the  following  sign  conventions:  (a)  northern 
latitudes  and  eastern  longitudes  are  considered  to  have  positive  signs  and  (b)  altitudes  above 
mean  sea  level  are  considered  positive. 

The  camera’s  position  is  first  expressed  in  geocentric  Cartesian  coordinates  as  follows. 
Let  if:,  X  and  h  be,  respectively,  latitude,  longitude  and  height.  Let  rg  be  the  radius  of  the 
earth.  Then,  from  Fig.  3,  the  camera’s  position  can  be  written  as 


Pa  = 


(re  +  h)  cos(^)  cos(A) 
(re  +  h)  cos(^)  sin(A) 
(re  +  h)  sin(V>) 


The  above  equation  is  used  to  obtain  the  position  Pg  of  the  camera  as  well  as  the  position 
Po  of  the  aim  point  in  the  site.  The  position  p„  of  the  camera  in  the  LVCS  can  then  be 
obtained  by  a  translation  and  two  rotations,  as 


Pui  —  Ib(a;,5r/2— ■!/')Ib(2,7r/2+A)(P5  Po) 


where 


cos(7r/2  +  A) 

sin(7r/2  +  A) 

0 

—  sin(A) 

cos(7r/2  +  A) 

0 

R(2:,7r/2-|-A)  — 

—  sin(7r/2  +  A) 

cos(7r/2  +  A) 

0 

—  cos(A) 

—  sin(A) 

0 

0 

0 

1 

0 

0 

1 
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1 

0 

0 

1 

0 

0 

0 

cos(7r/2  —  tp) 

sin(7r/2  —  xp) 

= 

0 

sin{xp) 

cos(^) 

0 

—  sin(7r/2  —  xp) 

cos(7r/2  —  xp) 

0 

—  cos{xp) 

sin('0) 

Using  target-centered  coordinates  The  aim  here  is  to  obtain  the  camera’s  position  in 
the  LVCS,  given  its  azimuth  elevation  a  and  range  r  w.r.t  the  target,  measured  as  shown 
in  Fig.  4.  We  choose  the  following  notation: 

•  ^  is  the  azimuth  angle  measured  in  the  McDonnell  Douglas  data  from  the  Y  axis 
clockwise  (i.e.  east  of  north).  Note  that  the  azimuth  measures  the  angle  between  the 
camera’s  pointing  direction  and  the  north.  To  determine  the  camera’s  position,  this 
azimuth  should  be  offset  by  tt. 

•  n  is  the  elevation  angle  measured  downwards  from  the  horizontal. 


r  cos(a)  cos(— 7r/2 

-p)' 

—r  cos(a)  sm{/3) 

P  = 

r  cos(a)  sin(— 7r/2 

—r  cos(q')  cos(/?) 

r  sin(a) 

r  sin(a) 

4  Computing  the  exterior  orientation  of  the  camera 

An  initial  exterior  orientation  of  the  camera  must  be  specified  in  the  conformal  photogram- 
metric  representation  in  the  RCDE.  We  describe  the  successive  transformations  leading  to 
this  representation. 

Attach  a  right  handed  local  vertical  coordinate  system  to  the  site  (LVCS)  with  the  Y 
axis  oriented  in  the  north  direction,  the  X  axis  pointing  toward  the  east  and  the  Z  axis  is 
in  the  positive  vertical  direction.  The  following  derivation  assumes  that  there  is  no  camera 
roll.  If  roll  is  small,  the  derivation  in  this  section  will  give  results  of  sufficient  accuracy  to 
perform  resection.  However,  if  roll  is  significant,  the  results  in  the  next  section  should  be 
used. 

In  order  to  align  the  LVCS  to  the  camera- centered  coordinate  system  (CCCS),  we  must 


successively  apply  the  following  axis  rotations: 
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1.  R(j:,_(0):  rotation  by  an  angle  ^  clockwise  around  the  vertical  Z  axis  to  align  the  Z-axis 
of  the  CCCS  with  the  YZ  plane  of  the  LVCS  (Fig.  5). 

2.  rotation  clockwise  around  the  resulting  X-axis  to  align  the  two  Z  axes 

(Pig.  6). 


We  have 


R 


{z-P)  — 


cos(/3)  —  sin(/3)  0 

sin(;d)  cos(/9)  0 

0  0  1 

1  0  0 

0  sin(Q!)  cos(a) 

0  —  cos(a)  sin(Q:) 


Then  R  =  R(a;,|_„)R(2,-/3),  i.e. 


cos(jd)  —  sin(jd)  0 

R=  I  sin(Q!)  sin(j5)  sin(Q;)  cos(;d)  cos(a:)  |  (1) 

—  cos(a)  sm(/3)  —  cos(a)  cos(jd)  sin(Q!) 

On  the  other  hand,  RCDE  uses  the  conformal  representation  to  relate  the  world  and 
camera  coordinate  systems.  In  this  case  the  relationship  between  (xjjj,  z^)  and  (zc,  J/c,  ^c) 
is  given  by 


\ 

Xc 

Vc 

= 

R^(ac)  Rj,(</>)  Rr(t^) 

Vw  -  Vo 

1 

o 

(  ■ 

cos  K  sm  K  U 

^  cos  <!>  0 

= 

—  sin  K  cos  K  0 

0  1 

(  »  °  h 

^  sin  </>  0 

1  0  0 

0  cos  u!  sin  u! 

0  —  sin  ct)  cos  LO 


-x^ 
Vw  -  Vo 
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5  Computing  roll 

After  performing  these  two  rotations,  the  Z  axis  and  the  XY  planes  of  the  two  coordinate 
systems  will  be  aligned.  A  further  rotation  around  the  Z-axis  is  required  to  account  for 
camera  roll.  The  determination  of  the  camera  roll  angle  7  (measured  clockwise  around  the 
Z-axis)  is  described  in  [3].  Once  this  angle  is  known,  a  final  transformation  is  needed, 

defined  by 

003(7)  sin(7)  0 

R-2,-7  =  —  sin(7)  003(7)  0 

0  0  1 

Therefore, 

005(7)  cos(yd)  +  sin(7)  sin(Q;)  sin(^)  —  003(7)  sin(/5)  +  sin(7)  sin(Q!)  cos(^)  sin(7)  cos(a) 
R=  —  sin(7)  cos(/?)  +  003(7)  sin(Q:)  sin(^)  sin(7)  sin(/3)  +  003(7)  sin(a)  cos(;3)  003(7)003(0) 

—  cos(q:)  sin(;0)  —  cos(a)  cos(/3)  sin(a) 
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Figure  7:  Measuring  control  point  locations  from  the  site  plan 


The  values  of  u)  and  (f)  are  the  same  as  before,  but  the  value  of  k  changes  to 


K  =  —Tan  ^ 


sin(7)  cos(/?)  +  cos(7)  sin(Q:)  sin(/9)  003(7)  cos(/d)  +  sin(7)  sin(Q:)  sin(^) ' 


—  cos(^) 


cos((f>) 


6  Obtaining  control  points  from  the  site  plan 

A  simple  scheme  for  obtaining  control  points  from  the  site  plan  is  illustrated  in  Fig.  7.  The 
following  are  the  steps  to  be  followed: 

1.  Choose  two  reference  points  on  the  plan  view,  lying  on  a  line  parallel  to  the  site  north 
direction,  which  is  the  principal  direction  in  which  the  buildings  are  oriented.  Measure 
the  distance  I  between  them. 

2.  Measure  ^1,^2,  and  Z  for  all  selected  control  points  {di  and  from  the  plan  view,  and 
Z  from  the  elevation).  Choose  the  origin  po  of  the  LVCS  as  one  of  the  control  points. 

3.  Convert  from  di,  d2  to  |X'|,  Y' .  The  relevant  formulae  are 

r'  =  Y,(4-4-e) 

\x'\  =  ,Jdi  - 


11 


(7) 

(8) 


4.  Equation  8  gives  only  the  absolute  value  of  X' .  Set  the  sign  of  X'  by  visual  inspection. 

5.  The  p  =  {X',  Y',  Z)  obtained  from  the  previous  steps  will  be  with  respect  to  a  coor¬ 
dinate  system  in  which  the  i/-axis  is  the  plant  north,  whereas  in  the  LVCS  the  ?/-axis 
is  (in  our  convention)  the  geographic  north,  at  an  angle  (f)  with  the  plant  north.  We 
therefore  need  to  rotate  the  original  coordinates  to  bring  them  into  the  LVCS.  Further, 
the  origin  po  be  shifted  from  the  actual  origin  of  the  LVCS  {po)t  s-nd  hence  we 
need  to  make  a  translation  to  bring  the  coordinates  po  to  (0,0,0).  For  any  point  p 
previously  measured,  the  conversion  is 


where 


and 


p  =  g(P-T) 


R  = 


cos[(j))  sin((;/i)  0 

—  sm(<f>)  cos{(f))  0 

0  0  1 


T  =  po 


7  Registration  and  Resection 

This  section  outlines  the  procedure  used  for  registering  images  to  a  site  model  using  RCDE. 
More  details  can  be  found  in  [3]. 

The  first  step  is  the  creation  of  a  “3-D  world”,  which  defines  the  LVCS  associated  with 
the  site.  The  origin  of  the  LVCS  is  chosen  to  be  some  central  point  in  the  site  for  which 
ground  truth  is  available.  To  register  an  image,  a  camera  object  is  created  and  associated 
with  it.  This  has  two  main  components,  internal  and  external  camera  parameters.  Internal 
parameters  are  focal  length  the  location  of  the  principal  point,  and  a  parameter  that  indicates 
the  type  of  projection  -  perspective,  orthographic  or  something  intermediate.^  These  are 
usually  available,  or  can  be  easily  determined.  External  parameters  are  the  position  and 

^RCDE  provides  a  more  sophisticated  method  for  camera  modeling,  called  the  FBIP  representation.  This 
method  is  applicable  in  cases  where  the  traditional  pinhole  model  is  not  accurate  enough. 
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Figure  8:  Example  of  resection:  on  the  left  we  show  the  site  model  projected  on  an  unregis¬ 
tered  image,  and  on  the  right  the  result  after  resection. 

orientation  of  the  camera  in  3-D.  These  parameters  are  determined  as  explained  in  Sections  3 
and  4.  However,  they  can  be  calculated  only  approximately,  and  are  not  accurate  enough 
to  be  used  directly  for  registration.  To  determine  accurate  external  camera  parameters, 
RCDE  provides  a  mechanism,  called  “resection”  in  photogrammetric  terminology,  which  is 
essentially  an  iterative  optimization  of  the  external  parameters  given  a  set  of  3-D  control 
points  and  their  locations  in  the  image.  The  idea  is  to  minimize  the  discrepancy  between 
the  image  positions  of  the  control  points  as  predicted  by  the  camera  model,  and  their  actual 
positions  as  observed  in  the  image.  The  approximate  parameters  determined  as  described 
earlier  are  used  to  initialize  the  optimization.  An  example  of  resection  is  shown  in  Fig.  8. 

8  Building  the  site  model 

In  this  section,  we  detail  the  methods  used  to  construct  a  site  model  in  RCDE,  given  a  set 
of  registered  reconnaissance  images  of  the  site.  It  is  assumed  that  a  plan  view  of  the  site  is 
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Figure  9:  Example  of  a  site  model:  the  Labadie  power  plant  site,  shown  from  two  different 
viewpoints.  Not  all  site  model  objects  are  shown. 

also  available  in  the  form  of  an  image,  and  that  this  image  has  been  registered  to  the  LVCS. 

Currently,  site  model  construction  is  done  manually.  Each  type  of  site  model  object  is 
handled  slightly  differently,  and  a  certain  amount  of  experience  is  needed  to  construct  a  site 
model  that  accords  well  with  the  reconnaissance  images.  Let  us  consider  the  most  common 
type  of  site  model  object,  the  rectangular  box  used  to  model  buildings.  A  box  is  defined 
by  its  size  parameters  (length,  breadth,  height),  as  well  as  its  location  and  orientation  in 
3-D.  RCDE  allows  the  creation  of  box  objects,  and  their  modification  by  varying  the  size, 
position  and  orientation  parameters.  Most  buildings  are  vertical,  and  their  images  in  the 
plan  view  appear  as  rectangles.  The  logical  place  to  start,  then,  is  the  plan  view  (Fig.  10). 
A  box  is  created,  and  its  horizontal  dimensions  and  position  are  modified  until  it  fits  the 
rectangular  outline  of  the  building  it  is  intended  to  model.  Now  we  look  at  a  different  image, 
and  see  whether  the  box  fits  the  building  in  it.  If  not,  we  make  the  necessary  adjustments  to 
the  box’s  parameters  until  a  good  fit  is  obtained.  This  procedure  is  then  repeated  for  a  few 


Figure  10:  Plan  view  of  the  Labadie  site,  with  the  site  model  superimposed. 

more  images  from  the  reconnaissance  set,  until  the  box  fits  the  building  equally  well  from 
all  views. 

Other  site  model  objects  of  interest  are  cylinders,  roads,  2-D  and  3-D  closed  curves,  etc. 
The  procedure  for  constructing  them  can  again  be  started  with  the  plan  view,  and  can  then 
be  refined  with  the  help  of  a  few  other  images.  More  details  on  site  model  construction  using 
RCDE  can  be  found  in  [3].  The  site  model  constructed  for  the  Labadie  power  plant  is  shown 
in  Fig.  9. 

9  Reference  image  generation 

Given  a  3-D  site  model,  images  of  the  site  can  be  synthesized  from  assumed  camera  posi¬ 
tions.  In  RCDE,  a  “sun  ray”  vector  is  first  created  for  the  site,  indicating  the  direction  of 
illumination.  A  rendered  image  can  then  be  generated  assuming  a  Lambertian  scattering 
model.  An  example  is  shown  in  Fig.  11. 
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Figure  11:  Example  of  rendering:  two  synthesized  views  of  the  Labadie  site. 

10  Future  Work 

The  work  described  so  far  has  been  implemented  and  tested  on  the  Labadie  data  set.  While 
the  results  are  satisfactory,  the  goal  of  terminal  platform  guidance  requires  the  acceleration 
and/or  automation  of  some  of  the  steps  involved.  We  plan  to  concentrate  on  the  following 
two  areas. 

10.1  Fast  site  model  construction 

Site  model  construction  using  RCDE  in  its  present  form  is  slow  and  laborious,  since  all  the 
work  is  done  manually.  A  semi-automatic  scheme  for  rapid  construction  of  specific  types  of 
site  model  objects  will  be  explored.  For  instance,  boxes  could  be  fit  to  rectangular  buildings 
automatically  by  means  of  an  optimization  algorithm  once  the  operator  has  identified  the 
corresponding  corners  in  the  reconnaissance  images  (for  instance,  see  [4]).  For  irregular 
objects  such  as  rivers  (represented  geometrically  by  2-D  or  3-D  ribbons),  the  operator  could 


16 


be  asked  to  locate  the  object  approximately  in  one  image.  An  active  contour  algorithm  could 
then  be  used  to  automatically  track  and  delineate  the  object. 

10.2  Dynamic  registration 

The  registration  of  the  IR  sequence  to  the  site  model  can  be  greatly  enhanced  by  using  a 
dynamic  model  for  the  motion  of  the  platform.  We  can  then  draw  upon  the  large  number 
of  recursive  techniques  that  have  been  developed  for  motion  analysis  (e.g.  [2]).  The  motion 
of  the  platform  is  assumed  to  be  smooth  to  an  arbitrary  order  (constant  velocity,  constant 
acceleration,  etc.).  A  state-space  representation  is  used  to  represent  the  time-evolution  of 
the  motion  of  the  platform  and  its  position  in  the  LVCS.  The  parameters  of  the  platform’s 
motion  constitute  the  elements  of  the  state  vector.  Using  the  dynamic  state  space  model, 
the  positions  of  salient  points  from  the  site  model  (for  instance,  building  corners)  can  be 
predicted  in  the  image,  and  their  true  positions  can  be  searched  for  within  a  small  search 
areas  centered  around  the  predicted  positions.  The  size  of  this  search  area  is  based  on  the 
uncertainty  in  the  prediction.  An  extended  Kalman  filter  is  used  to  estimate  the  motion 
parameters. 

An  interesting  feature  of  this  scheme  is  that  new  features  from  the  scene  can  be  tracked, 
and  their  positions  in  the  LVCS  automatically  estimated,  as  in  [2].  This  enables  the  auto¬ 
matic  refinement  of  the  site  model.  The  importance  of  this  is  that  the  original  site  model, 
based  on  images  taken  from  a  reconnaissance  aircraft  typically  flying  at  a  range  of  several 
kilometers  from  the  target,  loses  its  relevance  as  the  platform  approaches  the  target.  Fea¬ 
tures  from  the  original  site  model  disappear,  and  new,  “unknown”  features  appear  in  the 
image.  The  recursive  filtering  paradigm  enables  us  to  localize  these  new  features  and  add 
them  to  the  site  model. 
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